Skip to content

support lepton launching + add train stream script for esm2#1663

Open
jwilber wants to merge 2 commits into
mainfrom
jwilber/sae-lepton-ci
Open

support lepton launching + add train stream script for esm2#1663
jwilber wants to merge 2 commits into
mainfrom
jwilber/sae-lepton-ci

Conversation

@jwilber

@jwilber jwilber commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Adding lepton support for sae recipes.

Starting with ESM2 SAE, which I added producer/consumer support (train_streaming here) so no activation dataset needs to be extracted/saved during training.

@jwilber
support lepton launching + add train stream script for esm2
fac2214 
Signed-off-by: jwilber <jwilber@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cb47f875-0ca4-4b74-a76a-aa123d87ebb8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jwilber/sae-lepton-ci

Comment @coderabbitai help to get the list of available commands.

@jwilber jwilber self-assigned this Jun 25, 2026
@jwilber

jwilber commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 71e04fb

polinabinder1
polinabinder1 approved these changes Jun 29, 2026
View reviewed changes

@polinabinder1 polinabinder1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once checks pass

@trvachov

trvachov commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Is this the same pathway we do for other recipes integrations to Lepton? or this is a brand new pathway?

@jwilber

jwilber commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator Author

@trvachov Same pathway. the extra code you see here is because we don't want to save activations for CI, because those datasets are so large. so instead, we extract + train as we go, never materializing them to disk

@polinabinder1 polinabinder1 mentioned this pull request Jun 30, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@polinabinder1 polinabinder1 polinabinder1 approved these changes
@jstjohn jstjohn Awaiting requested review from jstjohn jstjohn is a code owner
@pstjohn pstjohn Awaiting requested review from pstjohn pstjohn is a code owner
@trvachov trvachov Awaiting requested review from trvachov trvachov is a code owner
@savitha-eng savitha-eng Awaiting requested review from savitha-eng savitha-eng is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@jwilber jwilber

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jwilber @trvachov @polinabinder1